Studying Political Behavior: A Comparison of Internet and Telephone Surveys

Despite the promise of Internet surveys, there are significant concerns about the representativeness of the sample and survey instrument effects. This article seeks to address these questions by examining the differences and similarities between parallel Internet and telephone surveys conducted in Quebec after the provincial election in 2007. Our results indicate that the responses obtained in each mode differ somewhat from each other but that few inferential differences would occur if conclusions were drawn from the analysis of one dataset or the other. We urge researchers to consider the Internet as a viable mode of data collection, in that the consequences of mode effects appear to be minimal. Conducting surveys is integral to the study of political behavior. Telephone survey methodologies made survey research more accessible by providing a less expensive and quicker alternative to face-to-face interviews. In recent years, however, telephone survey research has become more difficult, as response rates have fallen and technological advances have made it possible for individuals to avoid calls. The Internet has emerged as a potential alternative to the telephone for conducting survey research. Several polling firms (such as Zogby, Ipsos, Environics, and Léger Marketing) conduct online research projects, and the methodology has been used for the British Election Study and American Election Study. Despite its popularity, however, there has yet to be a definitive acceptance of Internet methodology as a tool of political behavior research. The British Election Study team has been testing the usefulness of the Internet survey mode in comparison to face-to-face interview methods since Correspondence concerning this article should be addressed to Laura B. Stephenson, Department of Political Science, University of Western Ontario, London, Ontario, Canada. E-mail: [email protected] by gest on Jauary 8, 2016 http://ijpoordjournals.org/ D ow nladed from 2001 (Sanders, Clarke, Stewart & Whiteley, 2002, 2007). In their 2007 article, they urge other researchers to investigate the usefulness of Internet surveys in other national contexts. This article takes up the challenge, and extends the study of Internet survey methodology in an important way—to consider how well such surveys compare with telephone surveys. Given that many surveys are conducted over the telephone, understanding how the two methodologies compare is crucial for understanding the value of Internet surveys. There are many reasons to think that neither the telephone nor Internet mode is perfect (Green & Krosnick, 2001; Malhotra & Krosnick, 2007), nor fully comparable to face-to-face interviews. However, the speed and flexibility that are possible with either telephone or Internet surveys have proven to be especially useful for certain studies of public opinion and elections, such as rolling cross-section surveys, and so both methodologies are valuable for researchers. To date, however, we simply do not know the comparability of the two methods. Although the debate has been quite active for some time in the United States (see, e.g., Couper, 2000 and Tourangeau, 2004) few studies have compared the use of Internet and telephone surveys outside the United States. Thus, this article not only expands the geographic range in the study of Internet surveys but also examines a different, but highly relevant, comparison. Despite the promise of Internet surveys, many researchers have significant and legitimate concerns about the representativeness of the sample and survey instrument effects. This article seeks to address these questions by examining the differences and similarities between parallel Internet and telephone surveys conducted in Quebec after the 2007 provincial election. As a political science methodology, the utility of Internet survey research has yet to be tested as a replacement for other tools. Thus, this project represents a significant advance for political science research into comparative politics, as well as a general contribution about the comparability of telephone and Internet surveys of public opinion. The Challenge Brady (2000) argues that like ‘‘telescopes in astronomy, microscopes in biology, and seismic, weather, and environmental sensors in the geo-sciences, surveys have features that make them a fundamental data collection method for the social sciences’’ (p. 47). Survey research is not a simple undertaking, however. Face-to-face interview surveys, the most traditional mode of surveying, are particularly onerous with respect to both the time and money involved in the research. Telephone survey methodologies are a quicker, less expensive, and acceptable alternative, although they do not yield data that is perfectly comparable to face-to-face interviews (Holbrook, Green & Krosnick, 2003). S T U D Y I N G P O L I T I C A L B E H A V I O R 25 by gest on Jauary 8, 2016 http://ijpoordjournals.org/ D ow nladed from However, using the telephone allowed for technological advances such as the rolling cross section method, which enables researchers to interview a number of individuals at time points throughout a campaign; the information gathered from this design opens up the possibility of evaluating campaign effects and the impact of various political events (Brady, Johnston, & Sides, 2006). The advent of computer-assisted telephone interview (CATI) technology further increased the value of this survey mode by making research techniques such as question rotation and question filtering possible, just as computer-assisted personal interviewing (CAPI) did for the traditional face-to-face interview method. Even though telephone survey research is widely used, it is not without flaws. The benefits of the method come at the cost of visual interviewing techniques that computer-assisted personal interviewing (CAPI) permits, or even the showing of question cards which is common in traditional face-to-face surveys. Telephone surveys have also been affected by lower response rates (Brehm, 1993; Tuckel & O’Neill, 2002) and new technologies that enable people to avoid surveyors by screening and/or blocking calls, or having unlisted numbers (Alvarez, Sherman, & Van Beselaere, 2003; Tourangeau, 2004). Alvarez et al. (2003) suggest that the recruitment difficulties that plaque telephone surveys ‘‘impl[y] that many RDD telephone surveys might be more error laden than is assumed by practitioners’’ (p. 28). As a response to this, and with the advent of new technology, the Internet has become an attractive alternative for researchers. Online samples have been used for many different purposes, from consumer studies (Braunsberger, Wybenga, & Gates, 2007) to election campaign polling (Healey & Hoek, 2000; Gibson & McAllister, 2008) to citizen surveys (Van Ryzin, 2008) to health studies (Spijkerman, Knibbe, Knoops, Van de Mheen, & Van den Eijnden, 2009). Internet surveys allow rapid data collection (Kennedy, Kuh, & Carini, 2000; Kaplowitz, Hadlock, & Levine, 2004; Fleming & Bowden, 2009) similar to telephone methodology, are relatively inexpensive to administer (Sanders, Clarke, Stewart, & Whiteley, 2007), and offer greater flexibility than telephone surveys with respect to the types of questions and information that can be included in the survey instrument (Berrens, Bohara, Jenkins-Smith, Silva, & Weimer, 2003). All of the techniques that were made possible with CATI technology remain possible with Internet surveys, and the visual nature of Internet surveys keeps open the range of possibilities for survey question design used with CAPI. This aspect of Internet surveys is especially attractive for experiments relating to methodological issues of survey design. Internet surveys also have other advantages to recommend them. Koch and Emrey (2001) find that purposive sampling of marginalized populations may be facilitated by the Internet, in that voluntary respondents to an online invitation to complete a survey were, demographically speaking, almost I N T E R N A T I O N A L J O U R N A L O F P U B L I C O P I N I O N R E S E A R C H 26 by gest on Jauary 8, 2016 http://ijpoordjournals.org/ D ow nladed from indistinguishable from the national sample of the marginalized group. Furthermore, when surveying elusive groups, Internet surveys offer researchers the ability to ask sensitive questions, which are more likely to be answered on the Internet rather than over the telephone (Coomber, 1997; Kreuter, Presser, & Tourangeau, 2008). For example, Link and Mokdad (2005, p. 241) found that in a comparison of Web and telephone surveys, a higher percentage of Web respondents reported heavy drinking. This relates to the findings of Berrens et al. (2003), who find that extreme positions are more likely to be taken in online surveys. Braunsberger et al. (2007) also suggest that Internet surveys produce more reliable data than telephone surveys. Other researchers have noted that for certain purposes, such as experimental surveys, Internet surveys are appropriate because ‘‘[t]rue probability samples may not be necessary to make valid inferences about relationships’’ (Berrens et al., 2003, p. 2). Finally, Internet surveys eliminate the possibility of interviewer effects (Braunsberger et al., 2007) and data entry errors (Fleming & Bowden, 2009). While the methodology is attractive for researchers, whether Internet surveys should be considered general substitutes for telephone surveys has yet to be established. The main issues to be addressed are the representativeness of Internet samples as compared to telephone samples, survey instrument effects, and the consequences of these effects for research conclusions. The concern about representativeness is fairly well-established in the literature (for an exception, see Bason, 2000). Couper (2000) and Spijkerman et al. (2009) caution that Internet surveys are vulnerable to coverage and sampling errors, and Smith (2001) argues that even the most representative Internet samples will still produce differences from traditional RDD telephone surveys. Concerns about coverage have decreased over the past decade as the percentage of homes with Internet access has grown substantially. Industry Canada’s website reports that since 1999, Internet penetration in industrialized nations has begun to approach telephony penetration. Using 2007 data, the report indicates that penetration in Canada was 65.9% and penetration in the United States was 71.7% (http://www.ic.gc.ca/eic/site/smt-gst.nsf/eng/sf09054 .html). However, other issues of representativeness remain. For example, Roster, Rogers, Albaum, and Klein (2004) found that telephone respondents were likely to be older than Internet respondents, and concerns about the demographic distribution of Internet use are well-documented in Couper (2000). It has also been argued that online samples are less representative because respondents are volunteers, who are likely to take part in a survey (or be part of an ongoing survey panel) because of a higher level of interest (Malhotra & Krosnick, 2007). Panel volunteers also tend to be more frequent Internet users than the general population (Vehovar, Lozar Manfreda & Batagelj, 2000). There is an additional concern that taking part in many surveys over time as part of a panel may adversely affect the quality of responses S T U D Y I N G P O L I T I C A L B E H A V I O R 27 by gest on Jauary 8, 2016 http://ijpoordjournals.org/ D ow nladed from (Van Ryzin, 2008), although there is some research to indicate that the responses of frequent panelists are more in line with actual behaviors (Coen, Lorch, & Piekarski, 2005). Many scholars have tried to address the representativeness of Internet surveys with weighting procedures. Taylor (1999/2000) finds that while demographic and propensity weighting significantly improve the comparability of telephone and Internet responses to political and attitudinal questions, the corrective is not perfect. The inability of weights to correct for sample differences is also found by Healey and Hoek (2000), who compared Internet, phone, and mail surveys. Schonlau et al. (2004) obtained similar results, but maintain that propensity weighting and post-stratification are promising strategies for overcoming differences. Berrens et al. (2003) argue that with appropriate weighting, each of the survey formats would lead to the same inferences on relational questions of interest to political scientists. Alvarez et al. (2003) echo this opinion, and note that the focus of social scientists should be on the relationships among variables. Thus, the effectiveness of weights in addressing issues of representativeness has yet to be resolved. A second concern about Internet surveys is that the data may be biased due to mode effects. Tourangeau (2004) notes that responses to survey questions are likely to differ by mode depending on the perceived level of privacy, legitimacy, and the cognitive burden of the question. Researchers looking into Internet mode effects have found several, but many of them recommend the method. Chang and Krosnick (2010) present evidence in favor of the use of Internet surveys by noting that computer surveys are less likely to be influenced by social desirability effects (answering based upon social norms) and satisfying (providing non-differentiating answers in order to minimize the cognitive effort and the time spent considering an answer) than oral interviews. Fricker, Galesic, Tourangeau, and Yan (2005), comparing telephone and Internet surveys, found that online respondents took longer to complete knowledge questions and answered a higher percentage correctly, recorded less non-response (due to prompting) and reported more consistent attitude responses (which they attribute to the visual component of Internet surveys). In addition, Roster et al. (2004) found that Internet survey responses were more likely to be negative or neutral than telephone responses, and that more extreme responses were reported. These findings are supported by the work of Survey firms have addressed these issues by refining their sampling methodology to make their Internet sample comparably representative to more traditional (face-to-face or telephone) sampling frames. In the United States, Knowledge Networks created a pool of internet respondents by making initial RDD telephone contact and then providing an internet connection to the household, which enables the company to overcome the obvious bias of resources and increases representativeness (Smith, 2001). Other companies, such as Harris Interactive and YouGov, use a mix of weighting and recruitment strategies to attempt to overcome the biases inherent in internet sampling. In Canada, Leger Marketing use a mix of online invitations and specific recruitment through telephone sampling to build a panel that is representative of the Canadian population. I N T E R N A T I O N A L J O U R N A L O F P U B L I C O P I N I O N R E S E A R C H 28 by gest on Jauary 8, 2016 http://ijpoordjournals.org/ D ow nladed from Kreuter et al. (2008), who find that web surveys produce more accurate responses than telephone surveys, as individuals are less affected by social desirability. As well, studies of the use of computers for survey administration (an essential aspect of Internet surveys) have shown that computers reduce the cognitive demands on respondents, thus mitigating some skill-level effects (Chang & Krosnick, 2010), and have been found to reduce the number of completion mistakes, blank items, and answer refusals (Kiesler & Sproull, 1986; see Fricker et al., 2005 for results with Internet surveys). Nantel and Lafrance (2006) go so far as to suggest that preliminary evidence indicates that web surveys are in fact superior to telephone surveys, in that they provide almost identical responses to many issues and differences, when found, can be explained by a lack of pressure to respond to answers when ‘‘don’t know’’ or ‘‘I do not wish to answer’’ is preferred. The most important question for researchers of political and electoral behavior, however, is whether these differences translate into substantive differences in terms of inferences. If the sample populations differ, but produce the same results, then the Internet may well be a viable alternative to the telephone for survey researchers. To the best of our knowledge, no telephone-Internet comparisons have been conducted with respect to electoral behavior. However, two published studies have compared Internet and face-to-face surveys in terms of election research, with contradictory results. Sanders and his colleagues included an Internet component on the 2005 British Election Study, and in their 2007 paper they present evidence that indicates substantive mode differences that have minimal consequences. In a side-by-side comparison with face-to-face interviews, they find that there are significant differences in the responses to questions in each survey, but very few significant mode effects in turnout and vote choice models. Malhotra and Krosnick (2007) examine this research question in the American context. They utilize four different surveys to investigate the differences between face-to-face and Internet studies—the 2000 American National Election Study (ANES), a 2000 Harris Interactive study, the 2004 ANES, and a 2004 YouGov Internet study. They find, similar to Sanders et al. (2007), that there are distinct differences in the samples in each year in terms of attitudes and characteristics. In contrast to Sanders et al., however, they find that there are significant inferential effects with respect to turnout and vote choice. They caution against using Internet samples, suggesting both that the sampling methodology (using volunteers) leads to lower accuracy than probability samples and that there are possible mode effects which could affect findings and generalizations about a population. Thus, while some evidence suggests that Internet surveys can be very effective and appropriate survey tools, issues of representativeness still cause many to be cautious about using Web surveys. Without a better understanding of its comparability to the widely accepted telephone mode, researchers may S T U D Y I N G P O L I T I C A L B E H A V I O R 29 by gest on Jauary 8, 2016 http://ijpoordjournals.org/ D ow nladed from continue to be reluctant to use the methodology. This article makes a contribution to the literature by providing just such an analysis. Hypotheses Given the extant literature, we begin our analysis with two broad, competing hypotheses. The first is that Internet surveys yield results that are comparable to telephone surveys for political and electoral behavior research. The second is that the differences between the two modes of survey research make the Internet an inappropriate alternative to telephone methodologies. To assess these two hypotheses, we consider two aspects of the data. First, how comparable are the responses in the two survey modes, and does weighting improve the comparability of the results? We expect that there will be some demographic differences between the Internet and telephone survey samples, but in light of the contradictory claims about the effect of weighting we have no specific expectations whether weighting the data improves the comparability to the general population. Second, given that we are interested in the utility of Internet surveys for political research, we consider whether the differences between the results (if they exist) translate into substantive differences in research conclusions. Building on the studies mentioned above, there is reason to expect that either hypothesis might be supported. Data The data used in this article were collected by Leger Marketing immediately after the 2007 Quebec provincial election. The respondents were adults (18 years of age and older) who were able to answer the survey in either French or English. The telephone survey was administered, using CATI, to a sample of 1003 respondents between 4 and 15 April. The probabilistic sample was proportionally stratified by region (Quebec is divided into five regions, Montreal CMA, Quebec CMA, West, Centre and East) and there was random selection of households within each region. The identical survey was administered over the Internet to a sample of 1172 respondents between 5 and 11 April. The Internet sample was drawn from a panel of more than 150,000 volunteers who are representative of the Quebec population. Individuals were recruited through omnibus surveys and email solicitations that were randomly sent to individuals identified through phone directories and the main Canadian Internet providers (prior to anti-spam rules). The sample that was drawn for this survey was, similar to the telephone survey, stratified by region (proportionally) and individuals were selected randomly within each stratum. The Internet sample is therefore a probabilistic sample of the entire Internet panel. Given the care taken in developing it, Leger’s online panel can be considered I N T E R N A T I O N A L J O U R N A L O F P U B L I C O P I N I O N R E S E A R C H 30 by gest on Jauary 8, 2016 http://ijpoordjournals.org/ D ow nladed from high quality among Internet panels, and thus likely to produce reliable data. In terms of expense, the Internet survey cost approximately two-thirds of the amount for the telephone survey. The response rate for the telephone survey was 50.9% (AAPOR RR1). The response rate for the Internet survey was much lower, 23.5% (AAPOR RR1; 5000 initial emails were sent). We are mindful that this may affect representativeness, as it indicates substantial self-selection into the survey. In terms of item nonresponse, we found that in general there were more refusals and reported ‘‘don’t know’’ answers (substantive nonresponse) in the Internet study, consistent with the findings of Braunsberger et al. (2007) and Roster et al. (2004). In very few cases, nonresponse was higher in the telephone survey than the Internet survey—two of the 65 questions analyzed in terms of refusals and seven of the 65 questions when considering ‘‘don’t know’’ responses. When considering both actual and substantive nonresponse, only 12 of the 65 questions considered had no significant differences across mode at p< .05 (see Table A1). To maximize our use of the data, in all of our analyses missing data were assigned mean or neutral values. In our conclusion, we consider how nonresponse may have affected our results. This study is ideal to research the comparability of Internet and telephone surveys for political and electoral behavior research for two reasons. First, the survey was administered, in both modes, by the same firm. The sampling techniques applied by Leger are similar for telephone and Internet studies, in that they are both stratified by region and are designed to be representative of the population of Quebec. Second, the surveys that were administered to the telephone and Internet samples were identical. Each questionnaire asked the same questions, including basic demographics, issue importance, partisanship, attitudes toward parties and leaders, and policy preferences. The questionnaire was designed by Éric Bélanger, Richard Nadeau, Brian Tanguay, and the authors as part of a larger study. Because the questionnaires were identical, the difficulties of finding comparable questions and dealing with potential question wording effects were alleviated. Results To get a complete picture of how Internet and telephone surveys compare, we analyze the data in several different ways. First, we report the differences in marginal distributions, and the significance of any differences. We then turn to understanding the substantive effects of the mode variation. We first consider bivariate relationships between voting for the incumbent party compared to all others (incumbent voting) and demographic, issue, and evaluation variables. Finally, we estimate an incumbent voting model that takes into account such independent variables, as a more realistic demonstration of the effects S T U D Y I N G P O L I T I C A L B E H A V I O R 31 by gest on Jauary 8, 2016 http://ijpoordjournals.org/ D ow nladed from likely to be found by researchers. Only those who reported voting were included in these analyses. As with other studies (Malhotra & Krosnick, 2007; Sanders et al., 2007) we look at weighted and unweighted data, using the demographic weights supplied by Leger. The weights were calculated by the survey firm. Within each region, a weight was created based on age, gender, and language (mother tongue), using the latest census data. Marginal Distributions The first point of comparison between telephone and Internet surveys is how closely the results resemble each other. Tables 1 and 2 report the mean or marginal values of several demographic and political variables in each sample, with and without weighting. Consider first the demographic variables in Table 1. In the unweighted sample, only two of the six demographic variables (gender and income) do not vary significantly between the two samples. In the weighted sample, the differences in age and language are corrected, although the Internet sample continues to have a higher mean level of education and lower church attendance. Turning to the political variables, the comparability across the two samples seems stronger, although not perfect. Even in the weighted sample, the proportion of individuals who identify with the provincial Action Démocratique du Québec (ADQ) remain significantly different between the two samples, as do the proportions of those that report voting for the federal New Democratic Party (NDP) and Green Party (PV). Turning to Table 2, what is immediately clear is that many real differences exist between the results from each sample. Using the unweighted data, only 12 of the 52 variables reported show no significant difference between the two samples; with weighted data, that number increases only slightly, to 16 of 52. In both analyses, fewer than half of the variables are indistinguishable from each other. Similar to the expectations of Berrens et al. (2003), and unlike the findings of Taylor (2000), weighting does little to increase comparability between the samples on attitudinal variables. Of course, the point of conducting a survey is to gather data that represents the true values of the variables in the target population. Given that we can access data about turnout and vote choice from official results (Directeur general des elections du Québec [DGEQ], 2008), we can see how closely our survey sample results resemble the general population of Quebec (Table 3). For the most part, our survey results for these political variables are undifferentiated by mode, whether the data is weighted or unweighted. There are only two measures for which the telephone sample differs significantly from the Internet sample—turnout and the proportion of votes for the Parti québécois (PQ), calculated with weighted data. In this case, weights actually worsen the comparability of the sample. I N T E R N A T I O N A L J O U R N A L O F P U B L I C O P I N I O N R E S E A R C H 32 by gest on Jauary 8, 2016 http://ijpoordjournals.org/ D ow nladed from